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Math homework is transforming at a rapid rate with the continuous advances in educational 
technology. Computer-based homework, in particular, is gaining popularity across a range of 
schools, with little empirical evidence on how to optimize student learning. In the current study, 
middle school students (N = 143) solved a set of challenging algebraic problems on a computer- 
based homework assignment and received (1) no feedback, (2) correct-answer feedback, (3) try- 
again feedback, or (4) explanation feedback after each problem. Feedback resulted in higher posttest 
scores than no feedback, and this was true regardless of feedback type. On transfer items, correct- 
answer feedback has positive effects for low-knowledge students, but neutral effects for higher- 
knowledge students. Results suggest the provision of basic feedback on computer-based homework 
can benefit novice students’ mathematics problem solving. 
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Introduction 

Modern advances in educational technology and increasing access to computers gives math 
teachers a wide range of tools for assigning homework and assessing students’ problem-solving 
skills. Indeed, intelligent tutor systems and computer-based homework are quickly gaining popularity 
and prevalence in math classrooms across the world, necessitating research that crosses the borders 
of math education, cognitive science, and educational technology. Many of these computer systems 
are designed to enhance student learning through the use of targeted problem solving and feedback. 
However, recent evidence suggests there may be potential consequences of providing feedback 
during problem solving, especially for learners with higher prior knowledge (e.g., Fyfe & Rittle- 
Johnson, 2016). The goal of the current research was to evaluate the effects of different types of 
feedback on computer-based algebra homework for middle-school students. The selection of algebra 
was motivated by the recognition that algebra is a gatekeeper to future educational and employment 
opportunities (Adelman, 2006), and by concerns about students’ inadequate understanding of and 
preparation in algebra (NMAP, 2008). 


Theoretical Framework 

In general, math education research supports the use of feedback during problem solving. Meta- 
analyses show that, on average, feedback has positive effects on learning outcomes relative to no 
feedback (Hattie & Timperley, 2007). Indeed, Hattie and Timperley (2007) claim that “feedback is 
one of the most powerful influences on learning and achievement” (p. 81). They even identify 
feedback as one of the top ten influences on student achievement, along with direct instruction and 
reciprocal teaching. However, the effects of feedback vary widely (Kluger & DeNisi, 1996) 
suggesting that certain types of feedback may be more effective than others. In fact, during 
mathematics problem solving, students can sometimes learn just as much, if not more, when no 
feedback is provided (Fyfe & Rittle-Johnson, 2016; Nussbaumer et al., 2008). 

A growing body of evidence suggests that some of the variability in feedback effects is due to 
students’ prior knowledge (e.g., Fyfe, Rittle-Johnson & DeCaro, 2012; Krause et al., 2009). 
Specifically, feedback often has strong, positive effects for students with lower prior knowledge, but 
neutral or negative effects for students with higher prior knowledge. 
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Theoretically, there are several reasons why feedback may hinder problem solving relative to no 
feedback. These reasons are related to the students’ perception of the feedback, including their 
affective and cognitive processes. For example, feedback may draw attention to the self and elicit 
affective reactions that interfere with learning (Kluger & DeNisi, 1996). For example, feedback on 
incorrect answers can produce ego-threat (i.e., a threat to one’s positive self-image), which may 
reduce one’s confidence or motivation to continue. Students with higher prior knowledge may be 
especially affected by ego-threat as they likely have some expectation of performing well. Another 
possibility is that feedback overloads cognitive resources simply by providing additional information 
that needs integrated with the students’ existing knowledge (Sweller, van Merrienboer, & Paas, 
1998). For example, feedback can disrupt a student’s internal processing of the task and ultimately 
hinder his ability to learn from it. 

One key factor to consider may be the type of feedback provided. Dempsey et al. (1993) outlined 
a hierarchy of feedback types based on the information provided: 


No feedback: provides no information about the student’s response. 

Verification feedback: informs the student if the response 1f correct or incorrect. 
Correct-answer feedback: informs the student what the correct answer is. 
Elaborated feedback: provides some explanation for why a response is correct. 
Try-again feedback: allows one or more additional attempts to try again. 


oe ie 


One possibility is that providing feedback with more information will have positive effects for 
both low- and high-knowledge students because it provides helpful information for moving forward. 
However, providing more information may also have consequences because it is more likely to 
overload cognitive resources. There is some consensus that effective feedback should at least provide 
the correct answer (Kluger & DeNisi, 1996). But, the benefits of extra information are less clear (see 
Mory, 2004), particularly in the realm of computer-based homework. 

Computer-based math homework has several potential advantages relative to traditional paper- 
and-pencil homework, including the provision of immediate feedback to students on their 
performance. The goal of the current study was to evaluate the effects of this feedback using a 
particular system, ASSISTments.org (Heffernan & Heffernan, 2014). ASSISTments is a computer 
tutor system that can provide scaffolds and feedback to assist student. The use of computer-based 
homework offers several advantages for understanding the effects of feedback. 

First, computer-based homework provides an ecologically valid context in which to evaluate the 
role of feedback on problem solving. Many prior feedback studies have been conducted in laboratory 
contexts in the presence of a researcher. Here, we test the effects of feedback in an authentic learning 
setting on homework assignments given to students by their teachers. Second, computer-based 
homework represents a learning setting that may reduce the negative effects of feedback. As 
mentioned above, one condition under which feedback may hinder learning is when it draws 
attention to the self as opposed to the task (Kluger & DeNisi, 1996). Attention on the self can invoke 
evaluations of one’s abilities that interfere with the task. Computer-generated feedback is often 
viewed as a less evaluative source of information than person-generated feedback (Karabenick & 
Knapp, 1988), and may help decrease attention on the self and increase attention on the task. 
Computer-based homework may also reduce overload of cognitive resources by giving students 
control over when and how they process the feedback. 


Current Study 
The current study tested the effect of different types of feedback for middle school students 
solving algebra problems on computer-based homework via the ASSISTments system. Teachers and 
students who were already using ASSISTments were recruited so that the system was familiar to the 
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students and part of their regular classroom experience. Students were assigned to receive no 
feedback, correct-answer feedback, explanation feedback, or try-again feedback. Based on previous 
research, feedback was predicted to interact with prior knowledge such that feedback would have 
stronger, positive effects on problem solving for students with lower prior knowledge than for 
students with higher prior knowledge. 


Method 


Participants 

All students from two sixth-grade teachers’ classrooms and two seventh-grade teachers’ 
classrooms were invited to participate. The teachers taught at three different schools and were using 
ASSISTments.org as part of their regular classroom experience. Of their 160 students, 17 students 
were not included in the study as they did not complete all required sessions. The final sample 
contained 143 students (65 in sixth-grade and 78 in seventh-grade). 


Materials 

All materials were presented using ASSISTments.org. The pretest included six algebraic 
equations to solve (see Table 1). There were four different problems types: ax + b=c, b + ax =c, a(x 
+ b) =c, and a(x + b) +c =d. The homework assignment contained two worked examples at the 
beginning to re-familiarize students with correct problem-solving solutions (see Figure | for an 
example). The remaining problems were equations for students to solve on their own. Students 
solved 12 or 16 problems (i.e., three or four of each type of problem presented on the pretest). 
Whether students solved 12 or 16 problems reflected natural variation in teacher preference as two 
teachers opted for the 16-problem assignment (n = 65 students) and two teachers requested a shorter 
12-problem assignment (” = 78 students). Percent correct scores were calculated for each student 
based on the number of items he or she was assigned. The posttest included eight equations to solve 
(see Table 1). Four items were isomorphic to the pretest problems (1.e., learning items) and the 
remaining four were challenge problems with novel problem structures (1.e., transfer items). Percent 
correct for each subscale was calculated. 


Table 1: Problems Presented on the Pretest and Posttest 


Pretest Posttest 
1. 2x+3=23 5x + 13 =73 
2. 10 + 5x = 30 3 + 6x = 99 
>. 3(x + 1)=9 7(x + 4) = 63 
4. 7(x+3)+2=51 2(x +3)+4=16 
5. 5(x +3)+14=64 x/2+3=13 
6. 8(x + 2) = 56 ACK 2) + 3K 2) = 35 
rs -- x/9 +31 = 34 
8. -- 6(x + 4) + 2(x +4) = 48 


Note. On the posttest, problems | through 4 are learning items and 5 through 8 are transfer items. 


Design and Procedure 

The experiment had a pretest-homework-posttest design. Students completed the pretest during 
class or at home. Within three school days, students completed the homework assignment on their 
own. For the homework assignment, students were randomly assigned to one of four conditions: no- 
feedback (n = 25), correct-answer feedback (n = 44), explanation feedback (n = 41), or try-again 
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feedback (n = 33). Finally, students completed the posttest. All teachers assigned the posttest the 
same day students finished the homework, but three teachers had students complete it in class and 
one teacher had students complete it at home. 


Example Problem 1 
Solve for x in the following equation: 
3x + 15 = 27 


Example Solution 
Here is one correct way to solve this problem. 


3x + 15 = 27 State the problem. 


3x + 15-15 = 27-15 Since 15 is added to 3x, subtract 15 from both sides of the equation. 
Simplify. 
Since x is multiplied by 3, divide both sides by 3. 


Now x is alone on one side of the equation. The solution is 4. 


Type the solution in the box below. 


Submit Answer | 


Figure 1. Worked example. An example given at the beginning of the homework assignment. 


In the no-feedback condition, students did not receive any feedback during the assignment. After 
submitting each answer, the computer provided an “answer recorded” message and students clicked a 
button to move on to the next problem. In the correct-answer feedback condition, students received 
immediate, correct-answer feedback after each problem. If students typed the correct answer, a green 
check mark appeared with the word “Correct!” If students typed an incorrect answer, a red X 
appeared along with the words, “The correct answer is __” (with the correct answer filled in). In the 
explanation feedback condition, the feedback message included the correct answer, an explanation of 
why it was correct, and a worked example. For example, for 3x + 12 = 24, the feedback message 
stated: 


The correct answer is 4 because when x is 4 both sides of the equal sign have the same amount. 
Let’s plug 4 in for x and simplify to show that both sides have the same amount. 


3x +12=24 


3*4+12=24 
12+12=24 
24=24 


In the try-again feedback condition, the feedback message stated, “Sorry, try again. _is not 
correct” (with the student’s answer filled in). Students could continue inputting responses until they 
entered the correct answer or they could obtain the correct answer by clicking on a button. 


Data Analysis 

To examine the impact of feedback and prior knowledge, regression analyses were used for each 
outcome measure. Condition was dummy coded with correct-answer feedback, explanation- 
feedback, and try-again feedback entered into the models (with no-feedback as the reference group). 
Thus, each regression model included three condition variables, pretest score (mean centered), and 
three condition by pretest score interactions. 
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Results 


Pretest 

On average, children solved 85% of the pretest problems correctly (SD = 23%). Scores did not 
differ significantly by condition, F(3, 139) = 1.09, p = .36. Performance tended to be similar across 
all six items with percent correct on each item ranging from 80% to 92%. 


Homework 

The regression predicting homework scores from condition and prior knowledge was significant, 
F(7, 135) = 11.64, p < .000, R’ = .38, but the only individual predictor to reach significance was the 
main effect of prior knowledge, B = 0.53, SE = 0.16, p = .002. Students with higher prior knowledge 
on the pretest exhibited higher homework scores than students with lower prior knowledge. There 
were no main effects of feedback or feedback by prior knowledge interactions, ps > .10. Thus, 
feedback had little effect on performance during the homework. 

However, qualitative evidence suggested that children were learning over the course of the 
assignment. For example, four problems were of the form a(x + b) = c. Only 77% of students solved 
the first of these problems correctly, but 88% solved the fourth of these problems correctly. 
Similarly, four problems were of the form a + bx = c. Only 80% of students solved the first one of 
these correctly, but 92% solved the final one correctly. Further, on earlier problems, errors tended to 
reflect common misconceptions about variables, whereas errors on later problems were more diverse, 
suggesting that students were at least attempting correct strategies later in the assignment. For 
example, the very first problem was 3x + 12 = 24. Two of the most common incorrect answers were 
1.6 or 9. Students added the coefficient (3) and the isolated number (12) to get 15x on the left side. 
Then, they either calculated 24 divided by 15 or 24 minus 15 to get their answer. However, one of 
the last problems was 8 + 2x = 40, and no student showed evidence of making the mistake of adding 
the 8 and the 2. Thus, students appeared to improve over the course of the homework assignment, but 
not differentially by condition. 


Posttest Learning Items 

Students did well on the posttest learning items, solving nearly 90% correct. Indeed, on 3 of the 4 
problems students were near mastery. The fourth problem, 2(x + 3) + 4 = 16, proved somewhat 
difficult with only 79% of students solving it correctly. Estimates from the regression predicting 
percent correct on the learning items are presented in Figure 2. The overall regression was 
significant, F(7, 135) = 9.22, p < .000, R’ = .32. There was a significant, positive effect of prior 
knowledge, B = 0.36, SE = 0.18, p = .04. However, there were also significant main effects of 
correct-answer feedback, B = 8.91, SE = 4.33, p = .04, and try-again feedback, B = 13.51, SE = 4.58, 
p = .004, as well as a marginal effect of explanation feedback, B = 7.40, SE = 4.35, p = .09. Although 
prior knowledge did not significantly interact with any feedback type, ps > .10, an examination of 
Figure 2 suggests that the effect of explanation feedback was only marginal because it was primarily 
effective for high-knowledge children, but not low-knowledge children. Descriptively, we also 
compared the percent of children at mastery by condition. Fewer children in the no-feedback 
condition (36%) solved all four learning items correctly compared to children who received correct- 
answer feedback (66%), explanation feedback (66%), and try-again feedback (72%). Thus, feedback 
boosted learning on the posttest relative to no feedback. 
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Posttest Learning 
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Figure 2. Posttest scores. Scores on the learning items by condition and prior knowledge. 
Unstandardized regression coefficients are plotted at +1 standard deviation from the mean. 


Posttest Transfer Items 

Students struggled on the posttest transfer items, solving only 64% correct on average. This is 
expected given that these were novel problem types, which students did not practice. Although all 
four problems were difficult, the problems with two separate terms in parentheses, like 4(x + 2) + 3(x 
+ 2) = 35, were the most challenging. Estimates from the regression predicting percent correct on the 
transfer items are presented in Figure 3. The overall regression was significant, F(7, 135) = 5.10, p< 
.000, R? = .21. There was a significant positive effect of prior knowledge, B = 1.12, SE = 0.33, p= 
.001, but no main effects of feedback condition, ps > .20. However, there was a significant 
interaction between prior knowledge and the no-feedback vs. correct-answer feedback contrast, B = - 
0.79, SE = 0.40, p = .05. To examine the interaction, pretest scores were centered at one standard 
deviation below the mean in one model (i.e., low prior knowledge) and one standard deviation above 
the mean in a separate model (i.e., high prior knowledge). For low-knowledge students, there was a 
significant positive effect of correct-answer feedback, B = 28.35, SE = 12.54, p = .03. The effects of 
explanation feedback, p = .14, and try-again feedback, p = .35, were positive, but not statistically 
significant. However, for high-knowledge students, there were negative, but non-significant effect of 
correct-answer feedback, p = .49, explanation feedback, p = .40, and try-again feedback, p = .40 (see 
Figure 3). Thus, the effects of feedback on transfer to novel problems depended on children’s prior 
knowledge on the pretest. Feedback (particularly correct-answer feedback) resulted in better transfer 
than no feedback for low-knowledge children. But, all three types of feedback resulted in slightly 
lower transfer scores relative to no feedback for high-knowledge children. 
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Posttest Transfer 
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Figure 3: Posttest scores. Scores on the transfer items by condition and prior knowledge. 
Unstandardized regression coefficients are plotted at +1 standard deviation from the mean. 


Discussion 

The goal of this study was to test the effects of feedback and prior knowledge on computer-based 
algebra homework using the ASSISTments system. Middle school students were assigned to receive 
no feedback, correct-answer feedback, explanation feedback, or try-again feedback during problem 
solving. After a single assignment, feedback resulted in higher posttest learning scores than no 
feedback, and this was true regardless of feedback type. On transfer items, feedback interacted with 
prior knowledge, such that correct-answer feedback has positive effects for low-knowledge students, 
but neutral effects for higher-knowledge students. 

The findings from the current study make at least three contributions to research in mathematics 
education. First, the results demonstrate the benefits of three different types of feedback on problem 
solving. Further, they suggest that providing additional information or attempts does not always 
increase the efficacy of feedback. In fact, basic correct-answer feedback resulted in the best transfer 
for low-knowledge students, suggesting a possible threshold account (Phye, 1979). That is, when 
more information or support is provided beyond what is needed, it does not provide any additional 
advantage. Second, the results indicate that the benefits of feedback are strong for low-knowledge 
students, but that high-knowledge students sometimes do just as well without feedback during 
problem solving. Indeed, on posttest transfer, high-knowledge students tended to exhibit higher 
scores in the no-feedback condition, which is consistent with recent research (Fyfe & Rittle-Johnson, 
2016; Krause et al., 2009). The third contribution is to introduce an exciting new method to conduct 
experimental research in an ecologically valid classroom. The ASSISTments project is a system 
bringing researchers and teachers together to better assist and assess student learning (Heffernan & 
Heffernan, 2014). 

Future research is needed to test different types and schedules of feedback that are more dynamic 
and that adjust based on the student response. Indeed, one of the benefits of computer-based 
homework is the possibility of adapting the provision of feedback to students’ needs. Further, more 
work is needed to enhance the provision of feedback for high-knowledge students. The high- 
knowledge students in the current study did not benefit from feedback, but still had room to grow. 
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One potential solution is to give high-knowledge students more control over the feedback (Hays et 
al., 2010), allowing them to skip unnecessary feedback and spend more time on challenge problems. 
Future studies should explore these and other possibilities. 

In general, the current research is at the border of mathematics education, cognitive science, and 
educational technology. The specific results highlight the power and variability of feedback effects 
on computer-based homework. On the one hand, researchers and educators can marvel that such 
basic feedback can improve problem solving for novice students. On the other hand, these results 
challenge the intuition that feedback is always helpful and suggest that some students, under certain 
circumstances, can do just as well without it. 
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